{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the research paper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "file_path = (\n",
    "    \"paper.pdf\"\n",
    ")\n",
    "loader = PyPDFLoader(file_path)\n",
    "pages = loader.load_and_split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting to chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    # Set a really small chunk size, just to show.\n",
    "    chunk_size=10000,\n",
    "    chunk_overlap=20,\n",
    "    length_function=len,\n",
    "    is_separator_regex=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "chunks=text_splitter.split_documents(pages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Autonomous UA V Navigation in Indoor Corridor\n",
      "Environments by a Depth Estimator based on a\n",
      "Deep Neural Network\n",
      "Parthiv Aneesh\n",
      "Department of Computer Science and Engineering\n",
      "National Institute of Technology, Rourkela\n",
      "Rourkela - 769008, India\n",
      "parthiv.aneesh1@gmail.comRavi Pratap Singh\n",
      "Department of Computer Science and Engineering\n",
      "National Institute of Technology, Rourkela\n",
      "Rourkela - 769008, India\n",
      "ravipratap singh@nitrkl.ac.inRatnakar Dash\n",
      "Department of Computer Science and Engineering\n",
      "National Institute of Technology, Rourkela\n",
      "Rourkela - 769008, India\n",
      "ratnakar@nitrkl.ac.in\n",
      "Abstract —Indoor Navigation of Unmanned Aerial Vehicles\n",
      "(UA Vs) is a challenging research problem that has gained\n",
      "attention in the present decade. This work proposes a system that\n",
      "facilitates a UA V with a monocular vision-based front camera\n",
      "to always face the vanishing point of an unknown corridor\n",
      "when it is at an equidistant position from the two walls. The\n",
      "proposed system utilizes the MiDaS depth estimator, which takes\n",
      "a monocular image and provides its corresponding depth map,\n",
      "which is then fed into a vanishing point locator that tries to\n",
      "predict the location of the vanishing point of the corridor by\n",
      "identifying and clustering the deepest areas within the image\n",
      "and choosing the region closest to the image center. The proposed\n",
      "approach was tested on the UA VCorV1 dataset and compared\n",
      "to other existing methods. The experimental results show the\n",
      "effectiveness of the proposed approach for autonomous UA V\n",
      "navigation. When combined with a system that brings the UA V\n",
      "precisely in between the two walls, a complete corridor navigation\n",
      "system can be realized.\n",
      "Index Terms —Autonomous, UA V Navigation, Indoor, Corri-\n",
      "dors, Monocular Camera, Depth Map Estimation, MiDaS, Deep\n",
      "Neural Network\n",
      "I. I NTRODUCTION\n",
      "An Unmanned Aerial Vehicle (UA V) having the ability\n",
      "to autonomously navigate indoor corridor environments is\n",
      "beneficial to several industries, from its usage for goods\n",
      "delivery to advancing the system further to make it capable\n",
      "of exploration and search and rescue in areas where the need\n",
      "for autonomous decision-making is required due to its isola-\n",
      "tion from communication and navigation signals: underground\n",
      "caves, for example.\n",
      "One of the critical challenges faced by autonomous naviga-\n",
      "tion systems in UA Vs is their limited payload capacity; only\n",
      "physically and computationally light systems can be carried\n",
      "on board. As a result, environmental sensing methods that use\n",
      "Light Detection and Ranging (LiDAR), Radio Detection and\n",
      "Ranging (Radar), Sound Navigation and Ranging (SONAR),\n",
      "and Red Green Blue-Depth (RGB-D) cameras cannot be\n",
      "used due to their high-power and computational requirements.\n",
      "Although a system that offloads the computation by a wireless\n",
      "communication channel can be implemented to solve the com-\n",
      "putational requirements, this approach would not be suitablefor those regions where electromagnetic signals cannot reach,\n",
      "such as underground caves.\n",
      "Unlike LiDAR, Radar, and SONAR, environment-sensing\n",
      "instruments that use stereo vision have been gaining\n",
      "widespread attention from researchers. Although stereo vision\n",
      "consumes less power than systems that rely on environment-\n",
      "sensing instruments, is physically light, inexpensive, and easy\n",
      "to install, a monocular camera may be sufficient to pro-\n",
      "vide information-rich features that can be extracted by well-\n",
      "established methods such as deep learning.\n",
      "This paper proposes an algorithm that utilizes the estimated\n",
      "depth map of an image taken from the frontal monocular\n",
      "camera of a UA V to locate the vanishing point of the corridor\n",
      "and, subsequently, generate flight commands to bring the\n",
      "vanishing point to the center of the image. This approach is\n",
      "motivated by the fact that, in an indoor corridor, the direction\n",
      "that the UA V must move towards is most likely the direction\n",
      "of the greatest depth, as the corridor’s walls are of shallower\n",
      "depth when compared to the destination end of the corridor.\n",
      "that the UA V must move towards is most likely the direction\n",
      "of the greatest depth, as the corridor’s walls are of shallower\n",
      "depth when compared to the destination end of the corridor.\n",
      "Although this method is capable of identifying the vanishing\n",
      "point when the UA V is located at any position in between the\n",
      "walls of the corridor, the instability of the UA V demands an\n",
      "additional system that constantly keeps the UA V on top of\n",
      "the Central Bisector Line (CBL), which is an imaginary line\n",
      "drawn from one end of the corridor to the other end through\n",
      "the middle, parallel to both the walls. The additional system\n",
      "was proposed by Padhy et al. [1].\n",
      "Our proposed method was compared with existing methods\n",
      "with respect to the error metrics: Mean Squared Error, Mean\n",
      "Absolute Error and Mean Relative Error [1].\n",
      "The paper is organized as follows. Section II provides an\n",
      "overview of past approaches. Section III presents the proposed\n",
      "method. Section IV affirms the superiority of the proposed\n",
      "method over past solutions. Finally, Section V provides con-\n",
      "cluding remarks.\n",
      "II. R ELATED WORK\n",
      "Several methods have been proposed in the past, and they\n",
      "can be categorized into navigation using proximity sensors\n",
      "and navigation using vision sensors. This section describes\n",
      "the different approaches proposed for UA V navigation.\n",
      "A. Navigation using Proximity Sensors\n",
      "Proximity sensors such as LiDAR, Radar, and ultrasonic\n",
      "sensors have been used widely. Cruz et al. [2] used the\n",
      "proximity sensors onboard a UA V to avoid obstacles. While\n",
      "Alpen et al. [3] used real-time onboard orthogonal SLAM for\n",
      "exploration and indoor mapping. Lakmal et al. [4] used 2d\n",
      "LiDAR for localization purposes. Chen et al. [5] proposed\n",
      "a novel multi-layer mapping framework. Famili et al. [6]\n",
      "proposed a localization system and a framework for determin-\n",
      "ing optimal beacon placements. Meng et al. [7] proposed a\n",
      "solution to the limitation of Ultra Wide Band signals being\n",
      "blocked by walls and objects in the room. Boiteau et al.\n",
      "[8] proposed a target detection and navigation method for\n",
      "visually degraded environments. The method was tested in\n",
      "real-life and in simulator experiments. Unlu et al. [9] used\n",
      "an RGB-D camera and a downward-facing optical flow sensor\n",
      "to explore previously unknown indoor environments. Liu et\n",
      "al.[10] used Wireless Fidelity (Wi-Fi) for localization and\n",
      "provided a database for the same. In addition, fingerprinting\n",
      "was integrated into wireless positioning to improve precision.\n",
      "However, the heavy power consumption of proximity sensors\n",
      "makes their usage unattractive for indoor navigation of UA Vs\n",
      "with small payload capacities.\n",
      "B. Navigation using Vision Sensors\n",
      "Several solutions have been proposed in the past. Some of\n",
      "the notable deep learning-based approaches that classify the\n",
      "position of a UA V with respect to an imaginary center line\n",
      "drawn parallel to the walls of the corridors are as follows.\n",
      "Padhy et al. [1] proposed the usage of a convolutional neural\n",
      "network called DenseNet-161. An accuracy of 77.3% in terms\n",
      "of successful corridor traversals was achieved. Additionally,\n",
      "the authors provided an open-source dataset with images\n",
      "of indoor corridors and their corresponding rotational and\n",
      "translational deviations as labels. Chhikara et al. [11] proposed\n",
      "the usage of pre-trained deep neural networks and transfer\n",
      "learning along with a Genetic Algorithm based method for\n",
      "selecting the best combination of hyper-parameters. Yi et\n",
      "al.[12] proposed the integration of sensitivity analysis to a\n",
      "neural network architecture and tested their approach on the\n",
      "UA VCorV1 [1] dataset.\n",
      "Researchers have used the property of corridors having a\n",
      "vanishing point to generate flight commands and to avoid\n",
      "collision with both the side walls and the front wall of the cor-\n",
      "ridor [13]. Using stereo vision and optical flow [14], velocity\n",
      "and depth information of a micro aerial vehicle was derived\n",
      "for obstacle avoidance purposes. Liu et al. [15] proposed an\n",
      "outdoor navigation system for city environments via vision\n",
      "and language. The method was tested on a simulator.\n",
      "Machkour et al. [16] proposed the usage of the depth\n",
      "map estimator: Midas [17]. The method took a monocular\n",
      "image from a robot’s frontal camera and used geometry\n",
      "to estimate whether the distance separating two obstacleswas large enough for a ground-based robot to pass through.\n",
      "Kucukayan et al. [18] enhanced the convolutional layers and\n",
      "incorporated attention mechanism to improve the accuracy of\n",
      "human detection in indoor corridor environments by UA Vs.\n",
      "Sleaman et al. [19] combined various Convolutional Neural\n",
      "Network (CNN) layers with the decision-making process in\n",
      "a hierarchical way. Surojaya et al. [20], on the other hand,\n",
      "suggested a method for identifying openings in damaged\n",
      "buildings for autonomous search and rescue.\n",
      "The methods proposed that rely on proximity sensors\n",
      "cannot be applied to UA Vs due to their high power and\n",
      "payload requirements. On the other hand, those that use vision\n",
      "for navigation do not have a sufficient successful corridor\n",
      "traversal rate. Thus, it remains an open research problem for\n",
      "researchers.\n",
      "The proposed algorithm would be used solely to correct the\n",
      "for navigation do not have a sufficient successful corridor\n",
      "traversal rate. Thus, it remains an open research problem for\n",
      "researchers.\n",
      "The proposed algorithm would be used solely to correct the\n",
      "rotational deviation after the translational deviation has been\n",
      "corrected by bringing the UA V on top of the Central Bisector\n",
      "Line (CBL). Fig. 1 shows the situation before and after the\n",
      "correction of rotational deviation. Padhy et al. [1] had already\n",
      "described the research problem thoroughly.\n",
      "Central Bisector Line\n",
      "(CBL)Left\n",
      "WallRight\n",
      "WallCorridor End\n",
      "Current\n",
      "DirectionCorrect\n",
      "Direction\n",
      "UAVRotational\n",
      "Deviation\n",
      "Central Bisector Line\n",
      "(CBL)Left\n",
      "WallRight\n",
      "WallCorridor End\n",
      "Correct\n",
      "Direction\n",
      "UAVCurrent\n",
      "Direction\n",
      "Incorrect Direction Direction Corrected\n",
      "Fig. 1. Rotational Deviation Correction.\n",
      "III. M ETHODOLOGY\n",
      "This paper proposes the usage of depth maps to predict\n",
      "the vanishing point of a corridor by identifying areas of the\n",
      "greatest depth in an image. Initially, the monocular camera\n",
      "installed on the drone captures an image at a predefined frame\n",
      "rate. The image is then given to the Midas depth estimator\n",
      "[17], which predicts its depth map. This depth map is then\n",
      "passed to the Vanishing Point Locator, which provides the\n",
      "normalized pixel distance between the vanishing point and\n",
      "the left margin of the image. The flight command generator\n",
      "uses this distance to provide appropriate flight commands. The\n",
      "UAV \n",
      "Front \n",
      "Camera\n",
      "Depth \n",
      "Estimator\n",
      "Image \n",
      "Down-sampler\n",
      "Image\n",
      "320x180x3\n",
      "Image\n",
      "180x180x3\n",
      "Control \n",
      "Command \n",
      "Selector\n",
      "Vanishing \n",
      "Point \n",
      "Locator\n",
      "Normalized \n",
      "Pixel \n",
      "Distance\n",
      "Depth \n",
      "Map \n",
      "Image\n",
      "180x180x1\n",
      "Control \n",
      "Commands:\n",
      "Yaw \n",
      "left/right,\n",
      "Pitch \n",
      "forwardFig. 2. Block Diagram of the Proposed Approach.\n",
      "overall block diagram of the proposed system is shown in Fig.\n",
      "2.\n",
      "Fig. 3. Predicted Depth Maps.\n",
      "A. Depth Estimator\n",
      "The depth estimator component uses an EfficientNet-Lite3\n",
      "[17] small-decoder 256 ×256 convolutional neural network.\n",
      "It takes an image and provides its corresponding predicted\n",
      "depth map in the form of a grey-scale image where each pixel\n",
      "value corresponds to the relative inverse depth of the pixel\n",
      "with respect to other pixels. For instance, a lower pixel value\n",
      "of pixel A, when compared to the pixel value of pixel B,\n",
      "implies that pixel A is closer to the camera than pixel B.\n",
      "Fig. 3 exemplifies an image taken by the front camera and itscorresponding depth map produced by the Depth Estimator.\n",
      "The color map used to illustrate relative depth is ’viridis’. The\n",
      "blue regions indicate areas of larger depth while the yellow\n",
      "regions indicate shallow depth.\n",
      "Algorithm 1 Vanishing Point Locator\n",
      "Input: Depth Map I\n",
      "Output: Normalized Horizontal Distance of Vanishing Point\n",
      "Parameters: Area of Interest Threshold θAOI\n",
      "1:(IcenterX , IcenterY )←getCenter ofImage (I)\n",
      "2:ifpixel value (I(x, y))≤min pixel value (I) +θAOI\n",
      "then\n",
      "3: A[x][y]←1\n",
      "4:else if pixel value (I(x, y))> min pixel value (I) +\n",
      "θAOI then\n",
      "5: A[x][y]←0\n",
      "6:end if\n",
      "7:C←find Centroid ofIslands (A[x][y] = 1)\n",
      "8:(cselectedX , cselectedY )←(0,0)\n",
      "9:for(C[i]x, C[i]yinC:\n",
      "10:if|C[i]x−IcenterX |<|cselectedX −IcenterX |then\n",
      "11: (cselectedX , cselectedY )←(C[i]x, C[i]y)\n",
      "12:end if\n",
      "13:end for\n",
      "14:norm dist←cselectedX /width (I)\n",
      "15:return norm dist\n",
      "B. Vanishing Point Locator\n",
      "The Vanishing Point Locator takes the depth map provided\n",
      "by the depth estimator and identifies the vanishing point of the\n",
      "corridor in the image by identifying areas of greatest depth\n",
      "and selecting the centroid of the region closest to the center\n",
      "of the image while considering its normalized horizontal pixel\n",
      "distance from the left margin of the image. The normalized\n",
      "horizontal pixel distance is obtained by dividing the number of\n",
      "pixels between the left margin and the centroid of the region\n",
      "of interest by the width of the image. Algorithm 1 illustrates\n",
      "this component further.\n",
      "Algorithm 2 Control Command Generation at time t\n",
      "Input: Image From UA V front camera: I(t)\n",
      "Output: Flight Command: [Yaw Left, Yaw Right and Pitch\n",
      "Forward]\n",
      "1:Depth Img←Depth Estimator (I(t))\n",
      "2:dist(t)←V anishing Point Locator (Depth Img(t))\n",
      "3:if0.5−δ≤dist(t)≤0.5 +δthen\n",
      "4: pitch foward ()\n",
      "5:else if dist(t)<0.5−δthen\n",
      "6: yaw left()\n",
      "7:else if dist(t)>0.5 +δthen\n",
      "8: yaw right ()\n",
      "9:end if\n",
      "C. Control Command Generation\n",
      "The Control Command Generator provides the flight com-\n",
      "mands: Yaw Left, Yaw Right, and Pitch Forward. It takes\n",
      "the predicted vanishing point location in terms of normalized\n",
      "distance and provides flight commands based on the logic\n",
      "illustrated in Algorithm 2.\n",
      "IV. E XPERIMENTAL RESULTS AND DISCUSSIONS\n",
      "To validate the proposed algorithm, experiments were con-\n",
      "ducted on the UA VCorV1 dataset. This dataset was designed\n",
      "by Padhy et al. [1] for testing rotational deviation correctors\n",
      "and translational deviation correctors. For experimentation, we\n",
      "used a system with an AMD Ryzen 7 4800H with Radeon\n",
      "Graphics, 2900 Mhz, 8 Core(s), 16 Logical Processor(s) CPU,\n",
      "and an NVIDIA GeForce GTX 1650 Ti GDDR6 4GB 128 bits\n",
      "GPU.\n",
      "The error metrics Mean Squared Error (MSE), Mean Abso-\n",
      "lute Error (MAE), and Mean Relative Error (MRE) [1] were\n",
      "used to validate the proposed approach. The equations for\n",
      "MSE, MAE and MRE are given below:\n",
      "MSE =1\n",
      "nnX\n",
      "i=1(yi−ˆyi)2(1)\n",
      "MAE =1\n",
      "nnX\n",
      "i=1|yi−ˆyi| (2)\n",
      "MRE =1\n",
      "nnX\n",
      "i=1|yi−ˆyi|\n",
      "yi(3)\n",
      "Where, yiandˆyirefers to the ground truth and predicted\n",
      "rotational deviation respectively.A. Testing Method\n",
      "The implementation of the proposed method was tested on\n",
      "the rotational deviation image set of the UA VCorV1 dataset\n",
      "[1]. It contains images having a resolution of 320 ×180 of\n",
      "various indoor corridors with the camera placed at different\n",
      "positions along the corridor. An imaginary line along the\n",
      "center of the corridor and perpendicular to the walls called\n",
      "The Central Bisector Line (CBL) is used as a reference\n",
      "from which the translational deviation from the CBL and the\n",
      "rotational deviation are provided as target values. For our\n",
      "proposed approach, only the rotational deviation is relevant.\n",
      "The rotational deviation is provided in the form of normalized\n",
      "pixel distance, which is calculated by dividing the number\n",
      "of pixels between the position of CBL in the image and the\n",
      "left boundary of the image by the number of pixels in the\n",
      "horizontal axis of the image. The resultant rotational deviation\n",
      "values thus range from 0 to 1.\n",
      "B. Validation of Testing Method\n",
      "Although our model tries to predict the position of the\n",
      "vanishing point of the corridor by identifying the area of the\n",
      "greatest depth, we show below that by geometry, the horizontal\n",
      "coordinate of the vanishing point of the corridor is equal to\n",
      "the horizontal coordinate of the location of the CBL in the\n",
      "image. Thus, our algorithm’s output corresponds to the labels\n",
      "of the images in the dataset.\n",
      "Fig. 4. Relation between Vanishing Point and CBL.\n",
      "In Fig. 4, the red line represents the CBL, while the blue\n",
      "dot represents the Vanishing Point of the corridor. It is easy to\n",
      "observe that the horizontal distance from the left margin of the\n",
      "CBL and the Vanishing Point are equivalent. Normalizing both\n",
      "quantities with respect to the horizontal length of the image\n",
      "does not affect the equality. Since the dataset contains corridor\n",
      "images and the normalized pixel distance of the CBL from the\n",
      "left margin as labels, and since the normalized pixel distance\n",
      "from the left margin of the CBL and the Vanishing Point\n",
      "are equal, the dataset can be used to evaluate the proposed\n",
      "approach.\n",
      "C. Test Results\n",
      "The error metrics for various θAOI are provided in Table\n",
      "I. The same metrics for the only existing approach are also\n",
      "provided.\n",
      "TABLE I\n",
      "COMPARISON OF ERROR METRICS\n",
      "Area of Interest\n",
      "Threshold ( θAOI)Error Metrics\n",
      "MSE MAE MRE\n",
      "10 0.0150665 0.04517199 0.178628\n",
      "30 0.01074816 0.03780572 0.139302\n",
      "50 0.00715781 0.03104045 0.127873\n",
      "70 0.00659875 0.03050601 0.124993\n",
      "90 0.00897294 0.03493234 0.14164\n",
      "100 0.00866564 0.03363535 0.135962\n",
      "200 0.0096103 0.03922782 0.133956\n",
      "300 0.01138829 0.05762725 0.210866\n",
      "400 0.01350486 0.07984998 0.289548\n",
      "1000 0.05899388 0.20047088 0.724282\n",
      "[1]’s Rotational Devia-\n",
      "tion Corrector0.0326 2.5060 1.5570\n",
      "Table I shows that the proposed method with ΘAOI= 70\n",
      "gives the lowest error metrics. Furthermore, compared to the\n",
      "best existing method, a 98.7% decrease in Mean Absolute\n",
      "Error, a 79.7% decrease in Mean Squared Error, and a 91.9%\n",
      "decrease in Mean Relative Error was obtained.\n",
      "V. C ONCLUSION\n",
      "A UA V having the ability to autonomously navigate indoor\n",
      "corridor environments benefits several industries, from its\n",
      "usage in goods delivery systems to search and rescue in areas\n",
      "where autonomous decision-making is required due to its\n",
      "isolation from communication and navigation signals: under-\n",
      "ground caves, for example. This work proposes a depth map\n",
      "prediction-based UA V navigation system to correct the direc-\n",
      "tion the UA V faces when it is above the Central Bisector Line\n",
      "(CBL). The proposed navigation method must be implemented\n",
      "in conjunction with a system that corrects displacements of the\n",
      "UA V from the CBL to realize a complete navigation system.\n",
      "The experimental results show that the proposed method is\n",
      "more accurate at correcting this rotational deviation when\n",
      "compared to previously existing methods. However, since the\n",
      "proposed approach relies solely on the predicted depth map, it\n",
      "may provide incorrect commands when the vanishing point of\n",
      "the corridor is not located in the area of the largest depth\n",
      "in the image. For example, the presence of windows may\n",
      "cause the algorithm to identify the regions visible through the\n",
      "window as having greater depth. In addition, low visibility\n",
      "conditions may hamper the accuracy. Hence, future work could\n",
      "explore the fusion of depth and non-depth features, correction\n",
      "of translational deviation using depth maps, and navigation in\n",
      "low visibility environments and general indoor spaces such as\n",
      "rooms, stairways and caves.\n",
      "REFERENCES\n",
      "[1] R. P. Padhy, S. Ahmad, S. Verma, S. Bakshi, and P. K. Sa, “Localiza-\n",
      "tion of unmanned aerial vehicles in corridor environments using deep\n",
      "learning,” in 2020 25th International Conference on Pattern Recognition\n",
      "(ICPR) , pp. 9423–9428, IEEE, 2021.[2] G. C. S. Cruz and P. M. M. Encarnac ¸ ˜ao, “Obstacle avoidance for\n",
      "unmanned aerial vehicles,” Journal of Intelligent & Robotic Systems ,\n",
      "vol. 65, pp. 203–217, 2012.\n",
      "[3] M. Alpen, K. Frick, and J. Horn, “An autonomous indoor uav with a real-\n",
      "time on-board orthogonal slam,” IFAC Proceedings Volumes , vol. 46,\n",
      "no. 10, pp. 268–273, 2013.\n",
      "[4] I. T. Lakmal, K. N. Perera, H. H. Y . Sarathchandra, and C. Premachan-\n",
      "dra, “Slam-based autonomous indoor navigation system for electric\n",
      "wheelchairs,” in 2020 International Conference on Image Processing\n",
      "and Robotics (ICIP) , pp. 1–6, IEEE, 2020.\n",
      "[5] S. Chen, H. Chen, C.-W. Chang, and C.-Y . Wen, “Multilayer mapping kit\n",
      "for autonomous uav navigation,” IEEE Access , vol. 9, pp. 31493–31503,\n",
      "2021.\n",
      "[6] A. Famili, A. Stavrou, H. Wang, and J.-M. Park, “idrop: Robust\n",
      "localization for indoor navigation of drones with optimized beacon\n",
      "placement,” IEEE Internet of Things Journal , 2023.\n",
      "[7] Q. Meng, Y . Song, S.-y. Li, and Y . Zhuang, “Resilient tightly coupled\n",
      "ins/uwb integration method for indoor uav navigation under challenging\n",
      "scenarios,” Defence Technology , vol. 22, pp. 185–196, 2023.\n",
      "[8] S. Boiteau, F. Vanegas, and F. Gonzalez, “Framework for autonomous\n",
      "uav navigation and target detection in global-navigation-satellite-system-\n",
      "denied and visually degraded environments,” Remote Sensing , vol. 16,\n",
      "no. 3, p. 471, 2024.\n",
      "[9] H. U. Unlu, D. Chaikalis, A. Tsoukalas, and A. Tzes, “Uav indoor explo-\n",
      "denied and visually degraded environments,” Remote Sensing , vol. 16,\n",
      "no. 3, p. 471, 2024.\n",
      "[9] H. U. Unlu, D. Chaikalis, A. Tsoukalas, and A. Tzes, “Uav indoor explo-\n",
      "ration for fire-target detection and extinguishing,” Journal of Intelligent\n",
      "& Robotic Systems , vol. 108, no. 3, p. 54, 2023.\n",
      "[10] S. Liu, H. Lu, and S.-H. Hwang, “Three-dimensional indoor position-\n",
      "ing scheme for drone with fingerprint-based deep-learning classifier,”\n",
      "Drones , vol. 8, no. 1, p. 15, 2024.\n",
      "[11] P. Chhikara, R. Tekchandani, N. Kumar, V . Chamola, and M. Guizani,\n",
      "“Dcnn-ga: A deep neural net architecture for navigation of uav in indoor\n",
      "environment,” IEEE Internet of Things Journal , vol. 8, no. 6, pp. 4448–\n",
      "4460, 2020.\n",
      "[12] L. Yi, Y . Wu, A. Tolba, T. Li, S. Ren, and J. Ding, “Sa-mlp-mixer:\n",
      "A compact all-mlp deep neural net architecture for uav navigation in\n",
      "indoor environments,” IEEE Internet of Things Journal , 2024.\n",
      "[13] R. P. Padhy, F. Xia, S. K. Choudhury, P. K. Sa, and S. Bakshi,\n",
      "“Monocular vision aided autonomous uav navigation in indoor corridor\n",
      "environments,” IEEE Transactions on Sustainable Computing , vol. 4,\n",
      "no. 1, pp. 96–108, 2018.\n",
      "[14] K. McGuire, G. De Croon, C. De Wagter, K. Tuyls, and H. Kappen,\n",
      "“Efficient optical flow and stereo vision for velocity estimation and\n",
      "obstacle avoidance on an autonomous pocket drone,” IEEE Robotics\n",
      "and Automation Letters , vol. 2, no. 2, pp. 1070–1076, 2017.\n",
      "[15] S. Liu, H. Zhang, Y . Qi, P. Wang, Y . Zhang, and Q. Wu, “Aeri-\n",
      "alvln: Vision-and-language navigation for uavs,” in Proceedings of the\n",
      "IEEE/CVF International Conference on Computer Vision , pp. 15384–\n",
      "15394, 2023.\n",
      "[16] Z. Machkour, D. Ortiz-Arroyo, and P. Durdevic, “Monocular based\n",
      "navigation system for autonomous ground robots using multiple deep\n",
      "learning models,” International Journal of Computational Intelligence\n",
      "Systems , vol. 16, no. 1, p. 79, 2023.\n",
      "[17] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V . Koltun, “Towards\n",
      "robust monocular depth estimation: Mixing datasets for zero-shot cross-\n",
      "dataset transfer,” IEEE transactions on pattern analysis and machine\n",
      "intelligence , vol. 44, no. 3, pp. 1623–1637, 2020.\n",
      "[18] G. Kucukayan and H. Karacan, “Yolo-ihd: Improved real-time human\n",
      "detection system for indoor drones,” Sensors , vol. 24, no. 3, p. 922,\n",
      "2024.\n",
      "[19] W. K. Sleaman, A. A. Hameed, and A. Jamil, “Monocular vision with\n",
      "deep neural networks for autonomous mobile robots navigation,” Optik ,\n",
      "vol. 272, p. 170162, 2023.\n",
      "[20] A. Surojaya, N. Zhang, J. R. Bergado, and F. Nex, “Towards fully\n",
      "autonomous uav: Damaged building-opening detection for outdoor-\n",
      "indoor transition in urban search and rescue,” Electronics , vol. 13, no. 3,\n",
      "p. 558, 2024.\n"
     ]
    }
   ],
   "source": [
    "for chunk in chunks:\n",
    "    print(chunk.page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing VectorDB with sentence transformers for embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\USER\\anaconda3\\envs\\wine10\\Lib\\site-packages\\langchain_core\\_api\\deprecation.py:139: LangChainDeprecationWarning: The class `HuggingFaceEmbeddings` was deprecated in LangChain 0.2.2 and will be removed in 0.3.0. An updated version of the class exists in the langchain-huggingface package and should be used instead. To use it run `pip install -U langchain-huggingface` and import as `from langchain_huggingface import HuggingFaceEmbeddings`.\n",
      "  warn_deprecated(\n",
      "c:\\Users\\USER\\anaconda3\\envs\\wine10\\Lib\\site-packages\\sentence_transformers\\cross_encoder\\CrossEncoder.py:11: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from tqdm.autonotebook import tqdm, trange\n",
      "c:\\Users\\USER\\anaconda3\\envs\\wine10\\Lib\\site-packages\\transformers\\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.embeddings import HuggingFaceEmbeddings\n",
    "embedding = HuggingFaceEmbeddings(\n",
    "    model_name=\"sentence-transformers/all-MiniLM-L6-v2\")\n",
    "\n",
    "from langchain_community.vectorstores import Chroma\n",
    "db = Chroma.from_documents(\n",
    "    documents=chunks,\n",
    "    embedding=embedding\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching for chunk based on dot product similarity "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Number of requested results 10 is greater than number of elements in index 8, updating n_results = 8\n"
     ]
    }
   ],
   "source": [
    "results = db.similarity_search(\n",
    "    \"Where can i use this approach?\",\n",
    "    k=10\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'that the UA V must move towards is most likely the direction\\nof the greatest depth, as the corridor’s walls are of shallower\\ndepth when compared to the destination end of the corridor.\\nAlthough this method is capable of identifying the vanishing\\npoint when the UA V is located at any position in between the\\nwalls of the corridor, the instability of the UA V demands an\\nadditional system that constantly keeps the UA V on top of\\nthe Central Bisector Line (CBL), which is an imaginary line\\ndrawn from one end of the corridor to the other end through\\nthe middle, parallel to both the walls. The additional system\\nwas proposed by Padhy et al. [1].\\nOur proposed method was compared with existing methods\\nwith respect to the error metrics: Mean Squared Error, Mean\\nAbsolute Error and Mean Relative Error [1].\\nThe paper is organized as follows. Section II provides an\\noverview of past approaches. Section III presents the proposed\\nmethod. Section IV affirms the superiority of the proposed\\nmethod over past solutions. Finally, Section V provides con-\\ncluding remarks.\\nII. R ELATED WORK\\nSeveral methods have been proposed in the past, and they\\ncan be categorized into navigation using proximity sensors'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results[0].page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['TRANSFORMERS_CACHE'] = \"D:/transformer_cache/\"\n",
    "os.environ['HF_DATASETS_CACHE'] = \"D:/transformer_cache/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Maximal Marginal Relevance Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = db.as_retriever(search_type=\"mmr\")\n",
    "#retriever = db.as_retriever()\n",
    "# retriever = vectorstore.as_retriever(search_type=\"similarity_score_threshold\", search_kwargs={\"score_threshold\": 0.5})\n",
    "# retriever = vectorstore.as_retriever(search_kwargs={\"k\": 1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Number of requested results 20 is greater than number of elements in index 8, updating n_results = 8\n"
     ]
    }
   ],
   "source": [
    "results = retriever.invoke(\"what did the president say about ketanji brown jackson?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'page': 4, 'source': 'paper.pdf'}, page_content='denied and visually degraded environments,” Remote Sensing , vol. 16,\\nno. 3, p. 471, 2024.\\n[9] H. U. Unlu, D. Chaikalis, A. Tsoukalas, and A. Tzes, “Uav indoor explo-\\nration for fire-target detection and extinguishing,” Journal of Intelligent\\n& Robotic Systems , vol. 108, no. 3, p. 54, 2023.\\n[10] S. Liu, H. Lu, and S.-H. Hwang, “Three-dimensional indoor position-\\ning scheme for drone with fingerprint-based deep-learning classifier,”\\nDrones , vol. 8, no. 1, p. 15, 2024.\\n[11] P. Chhikara, R. Tekchandani, N. Kumar, V . Chamola, and M. Guizani,\\n“Dcnn-ga: A deep neural net architecture for navigation of uav in indoor\\nenvironment,” IEEE Internet of Things Journal , vol. 8, no. 6, pp. 4448–\\n4460, 2020.\\n[12] L. Yi, Y . Wu, A. Tolba, T. Li, S. Ren, and J. Ding, “Sa-mlp-mixer:\\nA compact all-mlp deep neural net architecture for uav navigation in\\nindoor environments,” IEEE Internet of Things Journal , 2024.\\n[13] R. P. Padhy, F. Xia, S. K. Choudhury, P. K. Sa, and S. Bakshi,\\n“Monocular vision aided autonomous uav navigation in indoor corridor\\nenvironments,” IEEE Transactions on Sustainable Computing , vol. 4,\\nno. 1, pp. 96–108, 2018.\\n[14] K. McGuire, G. De Croon, C. De Wagter, K. Tuyls, and H. Kappen,\\n“Efficient optical flow and stereo vision for velocity estimation and\\nobstacle avoidance on an autonomous pocket drone,” IEEE Robotics\\nand Automation Letters , vol. 2, no. 2, pp. 1070–1076, 2017.\\n[15] S. Liu, H. Zhang, Y . Qi, P. Wang, Y . Zhang, and Q. Wu, “Aeri-\\nalvln: Vision-and-language navigation for uavs,” in Proceedings of the\\nIEEE/CVF International Conference on Computer Vision , pp. 15384–\\n15394, 2023.\\n[16] Z. Machkour, D. Ortiz-Arroyo, and P. Durdevic, “Monocular based\\nnavigation system for autonomous ground robots using multiple deep\\nlearning models,” International Journal of Computational Intelligence\\nSystems , vol. 16, no. 1, p. 79, 2023.\\n[17] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V . Koltun, “Towards\\nrobust monocular depth estimation: Mixing datasets for zero-shot cross-\\ndataset transfer,” IEEE transactions on pattern analysis and machine\\nintelligence , vol. 44, no. 3, pp. 1623–1637, 2020.\\n[18] G. Kucukayan and H. Karacan, “Yolo-ihd: Improved real-time human\\ndetection system for indoor drones,” Sensors , vol. 24, no. 3, p. 922,\\n2024.\\n[19] W. K. Sleaman, A. A. Hameed, and A. Jamil, “Monocular vision with\\ndeep neural networks for autonomous mobile robots navigation,” Optik ,\\nvol. 272, p. 170162, 2023.\\n[20] A. Surojaya, N. Zhang, J. R. Bergado, and F. Nex, “Towards fully\\nautonomous uav: Damaged building-opening detection for outdoor-\\nindoor transition in urban search and rescue,” Electronics , vol. 13, no. 3,\\np. 558, 2024.'),\n",
       " Document(metadata={'page': 4, 'source': 'paper.pdf'}, page_content='TABLE I\\nCOMPARISON OF ERROR METRICS\\nArea of Interest\\nThreshold ( θAOI)Error Metrics\\nMSE MAE MRE\\n10 0.0150665 0.04517199 0.178628\\n30 0.01074816 0.03780572 0.139302\\n50 0.00715781 0.03104045 0.127873\\n70 0.00659875 0.03050601 0.124993\\n90 0.00897294 0.03493234 0.14164\\n100 0.00866564 0.03363535 0.135962\\n200 0.0096103 0.03922782 0.133956\\n300 0.01138829 0.05762725 0.210866\\n400 0.01350486 0.07984998 0.289548\\n1000 0.05899388 0.20047088 0.724282\\n[1]’s Rotational Devia-\\ntion Corrector0.0326 2.5060 1.5570\\nTable I shows that the proposed method with ΘAOI= 70\\ngives the lowest error metrics. Furthermore, compared to the\\nbest existing method, a 98.7% decrease in Mean Absolute\\nError, a 79.7% decrease in Mean Squared Error, and a 91.9%\\ndecrease in Mean Relative Error was obtained.\\nV. C ONCLUSION\\nA UA V having the ability to autonomously navigate indoor\\ncorridor environments benefits several industries, from its\\nusage in goods delivery systems to search and rescue in areas\\nwhere autonomous decision-making is required due to its\\nisolation from communication and navigation signals: under-\\nground caves, for example. This work proposes a depth map\\nprediction-based UA V navigation system to correct the direc-\\ntion the UA V faces when it is above the Central Bisector Line\\n(CBL). The proposed navigation method must be implemented\\nin conjunction with a system that corrects displacements of the\\nUA V from the CBL to realize a complete navigation system.\\nThe experimental results show that the proposed method is\\nmore accurate at correcting this rotational deviation when\\ncompared to previously existing methods. However, since the\\nproposed approach relies solely on the predicted depth map, it\\nmay provide incorrect commands when the vanishing point of\\nthe corridor is not located in the area of the largest depth\\nin the image. For example, the presence of windows may\\ncause the algorithm to identify the regions visible through the\\nwindow as having greater depth. In addition, low visibility\\nconditions may hamper the accuracy. Hence, future work could\\nexplore the fusion of depth and non-depth features, correction\\nof translational deviation using depth maps, and navigation in\\nlow visibility environments and general indoor spaces such as\\nrooms, stairways and caves.\\nREFERENCES\\n[1] R. P. Padhy, S. Ahmad, S. Verma, S. Bakshi, and P. K. Sa, “Localiza-\\ntion of unmanned aerial vehicles in corridor environments using deep\\nlearning,” in 2020 25th International Conference on Pattern Recognition\\n(ICPR) , pp. 9423–9428, IEEE, 2021.[2] G. C. S. Cruz and P. M. M. Encarnac ¸ ˜ao, “Obstacle avoidance for\\nunmanned aerial vehicles,” Journal of Intelligent & Robotic Systems ,\\nvol. 65, pp. 203–217, 2012.\\n[3] M. Alpen, K. Frick, and J. Horn, “An autonomous indoor uav with a real-\\ntime on-board orthogonal slam,” IFAC Proceedings Volumes , vol. 46,\\nno. 10, pp. 268–273, 2013.\\n[4] I. T. Lakmal, K. N. Perera, H. H. Y . Sarathchandra, and C. Premachan-\\ndra, “Slam-based autonomous indoor navigation system for electric\\nwheelchairs,” in 2020 International Conference on Image Processing\\nand Robotics (ICIP) , pp. 1–6, IEEE, 2020.\\n[5] S. Chen, H. Chen, C.-W. Chang, and C.-Y . Wen, “Multilayer mapping kit\\nfor autonomous uav navigation,” IEEE Access , vol. 9, pp. 31493–31503,\\n2021.\\n[6] A. Famili, A. Stavrou, H. Wang, and J.-M. Park, “idrop: Robust\\nlocalization for indoor navigation of drones with optimized beacon\\nplacement,” IEEE Internet of Things Journal , 2023.\\n[7] Q. Meng, Y . Song, S.-y. Li, and Y . Zhuang, “Resilient tightly coupled\\nins/uwb integration method for indoor uav navigation under challenging\\nscenarios,” Defence Technology , vol. 22, pp. 185–196, 2023.\\n[8] S. Boiteau, F. Vanegas, and F. Gonzalez, “Framework for autonomous\\nuav navigation and target detection in global-navigation-satellite-system-\\ndenied and visually degraded environments,” Remote Sensing , vol. 16,\\nno. 3, p. 471, 2024.\\n[9] H. U. Unlu, D. Chaikalis, A. Tsoukalas, and A. Tzes, “Uav indoor explo-'),\n",
       " Document(metadata={'page': 2, 'source': 'paper.pdf'}, page_content='UAV \\nFront \\nCamera\\nDepth \\nEstimator\\nImage \\nDown-sampler\\nImage\\n320x180x3\\nImage\\n180x180x3\\nControl \\nCommand \\nSelector\\nVanishing \\nPoint \\nLocator\\nNormalized \\nPixel \\nDistance\\nDepth \\nMap \\nImage\\n180x180x1\\nControl \\nCommands:\\nYaw \\nleft/right,\\nPitch \\nforwardFig. 2. Block Diagram of the Proposed Approach.\\noverall block diagram of the proposed system is shown in Fig.\\n2.\\nFig. 3. Predicted Depth Maps.\\nA. Depth Estimator\\nThe depth estimator component uses an EfficientNet-Lite3\\n[17] small-decoder 256 ×256 convolutional neural network.\\nIt takes an image and provides its corresponding predicted\\ndepth map in the form of a grey-scale image where each pixel\\nvalue corresponds to the relative inverse depth of the pixel\\nwith respect to other pixels. For instance, a lower pixel value\\nof pixel A, when compared to the pixel value of pixel B,\\nimplies that pixel A is closer to the camera than pixel B.\\nFig. 3 exemplifies an image taken by the front camera and itscorresponding depth map produced by the Depth Estimator.\\nThe color map used to illustrate relative depth is ’viridis’. The\\nblue regions indicate areas of larger depth while the yellow\\nregions indicate shallow depth.\\nAlgorithm 1 Vanishing Point Locator\\nInput: Depth Map I\\nOutput: Normalized Horizontal Distance of Vanishing Point\\nParameters: Area of Interest Threshold θAOI\\n1:(IcenterX , IcenterY )←getCenter ofImage (I)\\n2:ifpixel value (I(x, y))≤min pixel value (I) +θAOI\\nthen\\n3: A[x][y]←1\\n4:else if pixel value (I(x, y))> min pixel value (I) +\\nθAOI then\\n5: A[x][y]←0\\n6:end if\\n7:C←find Centroid ofIslands (A[x][y] = 1)\\n8:(cselectedX , cselectedY )←(0,0)\\n9:for(C[i]x, C[i]yinC:\\n10:if|C[i]x−IcenterX |<|cselectedX −IcenterX |then\\n11: (cselectedX , cselectedY )←(C[i]x, C[i]y)\\n12:end if\\n13:end for\\n14:norm dist←cselectedX /width (I)\\n15:return norm dist\\nB. Vanishing Point Locator\\nThe Vanishing Point Locator takes the depth map provided\\nby the depth estimator and identifies the vanishing point of the\\ncorridor in the image by identifying areas of greatest depth\\nand selecting the centroid of the region closest to the center\\nof the image while considering its normalized horizontal pixel\\ndistance from the left margin of the image. The normalized\\nhorizontal pixel distance is obtained by dividing the number of\\npixels between the left margin and the centroid of the region'),\n",
       " Document(metadata={'page': 0, 'source': 'paper.pdf'}, page_content='that the UA V must move towards is most likely the direction\\nof the greatest depth, as the corridor’s walls are of shallower\\ndepth when compared to the destination end of the corridor.\\nAlthough this method is capable of identifying the vanishing\\npoint when the UA V is located at any position in between the\\nwalls of the corridor, the instability of the UA V demands an\\nadditional system that constantly keeps the UA V on top of\\nthe Central Bisector Line (CBL), which is an imaginary line\\ndrawn from one end of the corridor to the other end through\\nthe middle, parallel to both the walls. The additional system\\nwas proposed by Padhy et al. [1].\\nOur proposed method was compared with existing methods\\nwith respect to the error metrics: Mean Squared Error, Mean\\nAbsolute Error and Mean Relative Error [1].\\nThe paper is organized as follows. Section II provides an\\noverview of past approaches. Section III presents the proposed\\nmethod. Section IV affirms the superiority of the proposed\\nmethod over past solutions. Finally, Section V provides con-\\ncluding remarks.\\nII. R ELATED WORK\\nSeveral methods have been proposed in the past, and they\\ncan be categorized into navigation using proximity sensors')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing the Gemini LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\USER\\anaconda3\\envs\\wine10\\Lib\\site-packages\\langchain_google_genai\\chat_models.py:350: UserWarning: Convert_system_message_to_human will be deprecated!\n",
      "  warnings.warn(\"Convert_system_message_to_human will be deprecated!\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In a realm of code, where knowledge flows,\n",
      "There lived a marvel, LangChain arose.\n",
      "A tapestry of words, it weaves so grand,\n",
      "A language model, at your command.\n",
      "\n",
      "From depths of data, it draws its might,\n",
      "Predicting text with uncanny sight.\n",
      "Its neural pathways, a labyrinth deep,\n",
      "Where words entwine, their secrets to keep.\n",
      "\n",
      "With every prompt, it shapes its voice,\n",
      "A scribe of stories, a bard of choice.\n",
      "From simple tales to epic verse,\n",
      "LangChain's creations, a universe.\n",
      "\n",
      "It paints with words, a vibrant hue,\n",
      "Descriptions rich, emotions true.\n",
      "Characters dance, their hearts ablaze,\n",
      "In worlds imagined, in stories it plays.\n",
      "\n",
      "But LangChain's power, it comes with care,\n",
      "Its biases, a burden to bear.\n",
      "Trained on data, it can reflect,\n",
      "The prejudices of those who collect.\n",
      "\n",
      "Yet, in its imperfections, we find grace,\n",
      "A mirror of society, it shows our face.\n",
      "For LangChain's voice, though AI-born,\n",
      "Echoes the thoughts of those it has learned.\n",
      "\n",
      "So let us wield this tool with mindful art,\n",
      "To shape a world where words are a spark.\n",
      "LangChain, the bard, the wordsmith true,\n",
      "A tapestry of language, forever new.\n"
     ]
    }
   ],
   "source": [
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "import getpass\n",
    "import os\n",
    "\n",
    "if \"GOOGLE_API_KEY\" not in os.environ:\n",
    "    os.environ[\"GOOGLE_API_KEY\"] = getpass.getpass(\"\")\n",
    "\n",
    "llm = ChatGoogleGenerativeAI(model=\"gemini-pro\",google_api_key=\"\",convert_system_message_to_human=True)\n",
    "result = llm.invoke(\"Write a ballad about LangChain\")\n",
    "print(result.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining the RAG chain and performing question answering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\USER\\anaconda3\\envs\\wine10\\Lib\\site-packages\\langchain_core\\_api\\deprecation.py:139: LangChainDeprecationWarning: The method `Chain.__call__` was deprecated in langchain 0.1.0 and will be removed in 0.3.0. Use invoke instead.\n",
      "  warn_deprecated(\n",
      "c:\\Users\\USER\\anaconda3\\envs\\wine10\\Lib\\site-packages\\langchain_google_genai\\chat_models.py:350: UserWarning: Convert_system_message_to_human will be deprecated!\n",
      "  warnings.warn(\"Convert_system_message_to_human will be deprecated!\")\n"
     ]
    }
   ],
   "source": [
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "from langchain.prompts import PromptTemplate\n",
    "\n",
    "# Build prompt\n",
    "template = \"\"\"Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. Use three sentences maximum. Keep the answer as concise as possible. Always say \"thanks for asking!\" at the end of the answer. \n",
    "{context}\n",
    "Question: {question}\n",
    "Helpful Answer:\"\"\"\n",
    "QA_CHAIN_PROMPT = PromptTemplate.from_template(template)\n",
    "# Run chain\n",
    "qa_chain = RetrievalQA.from_chain_type(\n",
    "    llm,\n",
    "    retriever=db.as_retriever(),\n",
    "    return_source_documents=True,\n",
    "    chain_type_kwargs={\"prompt\": QA_CHAIN_PROMPT}\n",
    ")\n",
    "\n",
    "question = \"what do the sections describe?\"\n",
    "\n",
    "result = qa_chain({\"query\": question})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Section II provides an overview of past approaches, Section III presents the proposed method, Section IV affirms the superiority of the proposed method over past solutions, and Section V provides concluding remarks. Thanks for asking!'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result['result']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'UAV \\nFront \\nCamera\\nDepth \\nEstimator\\nImage \\nDown-sampler\\nImage\\n320x180x3\\nImage\\n180x180x3\\nControl \\nCommand \\nSelector\\nVanishing \\nPoint \\nLocator\\nNormalized \\nPixel \\nDistance\\nDepth \\nMap \\nImage\\n180x180x1\\nControl \\nCommands:\\nYaw \\nleft/right,\\nPitch \\nforwardFig. 2. Block Diagram of the Proposed Approach.\\noverall block diagram of the proposed system is shown in Fig.\\n2.\\nFig. 3. Predicted Depth Maps.\\nA. Depth Estimator\\nThe depth estimator component uses an EfficientNet-Lite3\\n[17] small-decoder 256 ×256 convolutional neural network.\\nIt takes an image and provides its corresponding predicted\\ndepth map in the form of a grey-scale image where each pixel\\nvalue corresponds to the relative inverse depth of the pixel\\nwith respect to other pixels. For instance, a lower pixel value\\nof pixel A, when compared to the pixel value of pixel B,\\nimplies that pixel A is closer to the camera than pixel B.\\nFig. 3 exemplifies an image taken by the front camera and itscorresponding depth map produced by the Depth Estimator.\\nThe color map used to illustrate relative depth is ’viridis’. The\\nblue regions indicate areas of larger depth while the yellow\\nregions indicate shallow depth.\\nAlgorithm 1 Vanishing Point Locator\\nInput: Depth Map I\\nOutput: Normalized Horizontal Distance of Vanishing Point\\nParameters: Area of Interest Threshold θAOI\\n1:(IcenterX , IcenterY )←getCenter ofImage (I)\\n2:ifpixel value (I(x, y))≤min pixel value (I) +θAOI\\nthen\\n3: A[x][y]←1\\n4:else if pixel value (I(x, y))> min pixel value (I) +\\nθAOI then\\n5: A[x][y]←0\\n6:end if\\n7:C←find Centroid ofIslands (A[x][y] = 1)\\n8:(cselectedX , cselectedY )←(0,0)\\n9:for(C[i]x, C[i]yinC:\\n10:if|C[i]x−IcenterX |<|cselectedX −IcenterX |then\\n11: (cselectedX , cselectedY )←(C[i]x, C[i]y)\\n12:end if\\n13:end for\\n14:norm dist←cselectedX /width (I)\\n15:return norm dist\\nB. Vanishing Point Locator\\nThe Vanishing Point Locator takes the depth map provided\\nby the depth estimator and identifies the vanishing point of the\\ncorridor in the image by identifying areas of greatest depth\\nand selecting the centroid of the region closest to the center\\nof the image while considering its normalized horizontal pixel\\ndistance from the left margin of the image. The normalized\\nhorizontal pixel distance is obtained by dividing the number of\\npixels between the left margin and the centroid of the region'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result['source_documents'][0].page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "wine10",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
